Automated storyboarding based on natural language processing and 2D/3D pre-visualization

ABSTRACT

Systems and methods are provided for a workflow framework that scriptwriters can utilize when developing scripts. A script can be parsed to identify one or more elements in a script, and various visual representations of the one or more elements and/or a scene characterized in the script can be automatically generated. A user may develop or edit the script which can be presented in a visual and temporal manner. Information parsed from the script can be stored in basic information elements, and used to create a knowledge bases.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims priority to U.S. Patent Application No.62/569,097, filed on Oct. 6, 2017, which is incorporated herein byreference in its entirety.

TECHNICAL FIELD

The present disclosure relates generally to storyboard creation, andmore particularly, to applying natural language processing (NLP)techniques to scripts in order to extract metadata upon whichstoryboards may be created. Moreover, the present disclosure relates totwo-dimensional (2D)/three-dimensional (3D) pre-visualization of one ormore aspects of a script.

DESCRIPTION OF THE RELATED ART

A storyboard can refer to a sequence of images or drawings that describetechnical and artistic parameters characterizing a form of media, suchas a movie, a performance, and the like. Examples of these parameterscan include a type of a shot, the position of a character, motion,camera, lighting, etc. that are used in creating a story on which themedia is based. Traditional production teams follow a process whereby ascript is developed or written. The script may then be translated intovisual representations by storyboard artists. Storyboards can be usefulfor visualizing a story, the timing of each character's appearance,exit, action(s), etc., and a corresponding environment while still inthe early stages of a media project, e.g., TV, live-action movie,animation, theatre, comics, novels, and other forms of media orperformances. Storyboard creation is usually an iterative and lengthyprocess performed by a group of storyboard artists that sketch thevisual representations by hand or with the help of digital sketchingsoftware.

Regarding the process of script writing, writers are often concernedwith maintaining logical consistency throughout a single story ormultiple, interrelated stories, such as those upon which movie series ormovie worlds are based. For example, a movie and subsequent sequels mayhave a magical theme or world as a backdrop that follows a set of rulesor parameters. To write scripts that are consistent with that magicaltheme or world, writers must focus on complying with those rules orparameters. Oftentimes, writers become preoccupied with the need tofollow such rules or parameters, and lose focus on story development.

BRIEF SUMMARY OF THE DISCLOSURE

In accordance with one embodiment, a computer-implemented methodcomprises presenting a visual representation of the script based on oneor more elements of a script, and presenting a visual representation ofan updated script based on one or more updates to the one or moreelements of the script. The computer-implemented method furthercomprises generating a visual representation of one or more characters'evolution in the script or updated script, and generating and updating,in real-time, a storyboard, based on the script or updated script,respectively, the storyboard including the visual representation of theone or more characters' evolution. Further still, thecomputer-implemented method comprises generating at least one of atwo-dimensional (2D), three-dimensional (3D), and camera-view preview ofthe script or updated script.

In one embodiment, the computer-implemented method further comprisespresenting a user interface receiving user input regarding at least anaddition or modification of one or more scene elements of the script orupdated script. In one embodiment, each of the one or more sceneelements corresponds to a location, dialog of the one or morecharacters, a transition, and an action. In accordance with oneembodiment, the computer-implemented method further comprises receiving,via the user interface, user input regarding a camera-shot optionassociated with the one or more scene elements corresponding to alocation, dialog of the one or more characters, and an action.

In one embodiment, the computer-implemented method further comprisesrecording, through the user interface, a soundtrack commensurate withthe dialog of the one or more characters.

In one embodiment, the generating and updating of the storyboardcomprises creating a storyboard frame based on each of the one or morescene elements.

In one embodiment, the visual representation of the one or morecharacters' evolution in the script or updated script comprises a 2Dtimeline view. In one embodiment, the 2D timeline view comprises atleast two axes, a first of the at least two axes representing the one ormore characters, and a second of the at least two axes representing atemporal aspect of the script or updated script. In one embodiment, eachof the one or more characters are represented by a timeline along thefirst of the at least two axes, and wherein interaction between at leasttwo of the one or more characters is visualized by a meeting of eachtimeline representative of the at least two of the one or morecharacters.

In one embodiment, the computer-implemented method further comprisespresenting one or more representations of one or more cameras, whereinuse of the one or more cameras relative to the one or more charactersspatially and temporally is reflected.

In one embodiment, the presentation of the 2D preview of the 3D previewcomprises presenting a 2D map or a 3D map including one or more elementsrepresented within the script or updated script, and a spatialrepresentation of the one or more elements set forth within the scriptor updated script.

In one embodiment, the computer-implemented method further comprisescorrelating one or more images associated with camera shots to the oneor more elements of the script or updated script to generate or updatethe storyboard.

In one embodiment, the computer-implemented method further comprisesgenerating a heating map in conjunction with at least one of the 2Dpreview of the 3D preview.

In one embodiment, the heating map represents a relationship between twoor more scene elements of the script or updated script.

In accordance with another embodiment, a system comprises at least oneof natural language processor and an analytics engine extractingmetadata from a script or an updated script. The system furthercomprises a visualization engine: presenting a visual representation ofthe script based on the metadata, the metadata being representative ofone or more elements of a script; presenting a visual representation ofan updated script based on one or more updates to the one or moreelements of the script; generating a visual representation of one ormore characters' evolution in the script or updated script; generatingand updating, in real-time, a storyboard, based on the script or updatedscript, respectively, the storyboard including the visual representationof the one or more characters' evolution; and generating at least one ofa two-dimensional (2D), three-dimensional (3D), and camera-view previewof the script or updated script.

In one embodiment, the at least one of the natural language processorand the analytics engine generates a knowledge base comprisinginformation inferred from the metadata, and at least one of rulesmetadata and user queries regarding the script or updated script.

In one embodiment, the system further comprises a user interfacereceiving user input regarding at least an addition or modification ofone or more scene elements of the script or updated script, each of theone or more scene elements corresponding to a location, dialog of theone or more characters, a transition, and an action.

In one embodiment, the visualization engine generates and updates thestoryboard by creating a storyboard frame based on each of the one ormore scene elements. In one embodiment, the visualization enginerepresents the one or more characters' evolution in the script orupdated script with a 2D timeline view. In one embodiment, the 2Dtimeline view comprises at least two axes, a first of the at least twoaxes representing the one or more characters, and a second of the atleast two axes representing a temporal aspect of the script or updatedscript.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure, in accordance with one or more variousembodiments, is described in detail with reference to the followingfigures. The figures are provided for purposes of illustration only andmerely depict typical or example embodiments.

FIG. 1 is a flowchart of example operations that can be performed toprovide a script workflow and generate one or more visualizations, suchas a 2D visualization, a storyboard, and/or a 3D pre-visualization inaccordance with various embodiments.

FIG. 2 is a schematic representation of a system architecture in whichthe workflow and visual generation operations of FIG. 1 may beimplemented.

FIGS. 3A-3L illustrate example functionality provided by a userinterface of a platform adapted to perform the workflow and visualgeneration operations of FIG. 1.

FIG. 4 is a schematic representation of a virtual reality datacontrol/user interface system architecture in accordance with variousembodiments.

FIG. 5A is an example of a parse request message to an externalprocessing component in accordance with various embodiments.

FIG. 5B is an example of a parse result message in response to the parserequest message of FIG. 10A.

FIG. 6A is an example of a parse interaction group request message to anexternal processing component in accordance with various embodiments.

FIG. 6B is an example of a parse interaction group result message inresponse to the parse interaction group request message of FIG. 11A.

FIG. 7 is an example implementation of a virtual reality data controlinterface in accordance with various embodiments.

FIG. 8 is an example architecture/workflow for natural languageprocessing in accordance with various embodiments.

FIG. 9 is an example of rule and error definition in accordance withvarious embodiments.

FIG. 10 is an example of natural language processing-based scriptdevelopment that includes rule and error definitions in accordance withvarious embodiments.

FIG. 11 illustrates example functionality of a visualization panelassociated with group interaction during script development inaccordance with various embodiments.

FIG. 12 is an example computing component that may be used to implementvarious features of embodiments described in the present disclosure.

The figures are not exhaustive and do not limit the present disclosureto the precise form disclosed.

DETAILED DESCRIPTION

Various embodiments are directed to a platform or set of tools withwhich a user may create or edit a script using various visualrepresentations, automatically determine thematic relationships, enforcethematic rules/parameters, as well as create a storyboard. Moreover, theplatform can provide a pre-visualization of a story. Users can bedirectors, scriptwriters, storyboard artists, and/or other usersinvolved in the creative process. In some embodiments, the platform mayinclude various components that alone, or in conjunction with each otherprovide a script creation or analysis/processing function. Anotherfunction is a timeline function that presents a graphical representationof how a character evolves within a story or scene as a function of ascript. Another function is a storyboard function that allows a user togenerate and/or update a script. Still another function of the platformincludes the aforementioned pre-visualization that allows a scene from ascript to be viewed by a user in real or near real-time, e.g., a runningpreview of the script as it is developed using the platform.

FIG. 1 is a flowchart of example operations that can be performed toprovide a script workflow and generate one or more visualizations, suchas a 2D visualization, a storyboard, and/or a 3D pre-visualization inaccordance with various embodiments. FIG. 1 may be described inconjunction with FIG. 2 which is a schematic representation of theworkflow and visual element generation of FIG. 1. At operation 100, oneor more elements of a script are received, and a visual representationof the script is presented. A user may enter a script through or usingfrontend application/UI 202 (FIG. 2) either by typing the script in orby uploading a script into frontend application/UI 202. The script maybe in a text format, and can be displayed on editing panel 204. Variousembodiments can present or display the script in editing panel 204 indifferent ways, e.g., a traditional, linear script layout withadditional features to assist in script writing.

Frontend analytics engine 206 and/or backend analytics engine 212 may beused to parse the script. Frontend analytics engine 206 and backendanalytics engine 212 may be resident on the same device or network, orthey may be remotely located from each other. For example, frontendanalytics engine 206 may be resident on a user device on which frontendapplication/UI 202 is implemented, while backend analytics may beimplemented on a remotely located backend server 210, and connected tofrontend application/UI 202 via one or more networks. The network may beany communications network such as a cellular or data network, asatellite network, an intranet, an extranet, a virtual private network(VPN), a local area network (LAN), a wireless LAN (WLAN), a wide areanetwork (WAN), a personal area network (PAN), a portion of the Internet,a portion of the Public Switched Telephone Network (PSTN), or anycombination thereof.

Depending on the complexity of the script, the type of analysis to beperformed, and/or the processing power of a device or processor withwhich the frontend application/UI 202 is implemented, a particularanalytics engine may be selected to operate on the script.

For example, frontend analytics engine 206 may be utilized to extractmetadata from the script. That is, frontend analytics engine 206 may beprogrammed with natural language processing functionality such that itcan analyze the text of a script and determine the existence ofmeaningful language, such as language indicative of characters, actions,interactions between characters, dialog, etc. For example, frontendanalytics engine 206 may extract metadata indicative of a character,e.g., based on text determined or known to be a name. Frontend analyticsengine 206 may extract metadata indicative of an action, e.g., based ontext indicative of action, such as verbs. Frontend analytics engine 206may extract metadata indicative of location, e.g., based on known namesof locations (geographical and/or set location) or other textuallocation information found in the script.

However, backend analytics engine 212 may also be utilized to parse thescript. Thus, backend analytics engine 212 may be used as a secondarycheck on the parsing performed by frontend analytics engine 206. Forexample, if the script contains large amounts of complex metadata, andbackend analytics engine 212 can parse the script faster, backendanalytics engine 212 may be utilized for parsing. If the script is in alanguage recognized only by backend analytics engine 212, again, backendanalytics engine 212 may be utilized instead of frontend analyticsengine 212. In some embodiments, frontend analytics engine 206 may beutilized to parse a script, while backend analytics engine 212 may beused to perform more intensive “story analytics” by checking for logicalconsistency vis-à-vis a compiler component 214 of backend analyticsengine 212. In some embodiments, parsing can be relegated to an externalcomponent or system, such as natural language processing (NLP) core 224(described in greater detail below), e.g., when processing requirementsand/or speed necessitate a dedicated NLP system.

In order to train frontend analytics engine 206 and/or backend analyticsengine 212, a logical, formal language can be developed using knownwords or most-used words to represent or indicate a character, location,action, etc. Using the natural language processing functionality,frontend analytics engine 206 and/or backend analytics engine 212 maydiscover and/or provide information that can be used by visualizationengine 208 to generate a visualization of the script (described ingreater detail below).

Referring back to FIG. 1, at operation 102, one or more updates to theone or more elements of the script are received and a visualrepresentation of the updated script is presented. Accordingly, frontendanalytics engine 206, backend analytics engine 212, and/or NLP core 224may perform the aforementioned processing to parse and analyze theupdated script or updated elements of the script.

Additionally, using the natural language processing functionality,frontend analytics engine 206, backend analytics engine 212, and/or NLPcore 224 may build relationships between the extracted metadatacommensurate with the logical, formal language developed in accordancewith various embodiments. For example, backend analytics engine 212 mayparse a script and determine that two characters A and B are present ina current scene. Based on additional extracted metadata indicative ofsome action in which characters A and B are engaged, determinations canbe made/predicted regarding characters' emotional states, positions,relationships, etc. As alluded to above, various embodiments may compilea script(s) and check for logical inconsistencies. Determinationsregarding whether or not logical inconsistencies exist may be determinedon this extracted metadata which is used to build a contextual “world.”Parsing in accordance with various embodiments can allow a visualizationto be realized from a script as well as allow for a compilation to bechecked and/or provide VR-related production suggestions.

It should be noted that both frontend/backend analytics engines 206/212and NLP core 224 may access historical information regarding one or moreaspects of the script or known information associated with the metadatain order to further enhance their ability to build the contextual world.For example, one or more persistent data stores (an example of which maybe embodied as a persistent data library 220) containing informationregarding a character's action and spoken history in previously producedmedia content that, e.g., the introduction of the character in acurrently produced VR experience is contextually consistent. As will bediscussed further below, persistent data library 220 may contain furtherinformation that can be leveraged in accordance with other embodiments.

It should be noted that backend server 210 may comprise a scriptsynchronization engine 216 and script storage 218, which may be one ormore databases or data repositories. Script synchronization engine 216may operate to synchronize one or more iterations of a script. Forexample, frontend application/UI 202 may be a web-based or standaloneapplication that can be implemented on one or more devicessimultaneously. This allows a user to develop a script using differentdevices. Script storage 218 may be used to store different iterations orversions of the script from the different devices. Scriptsynchronization engine 216 may access different iterations or versionsof a script in progress, compare the different iterations or versions,and update each accordingly so that the user is able to continuedevelopment of the script on any of his/her devices.

Referring back to FIG. 1, at operation 104, a knowledge base comprisinginformation inferred from the script or updated script and at least oneof rules metadata and user queries regarding the script or updatedscript is generated. In some embodiments (described below), informationencapsulated in the script or updated script is represented usingknowledge elements. Additionally, character systems (also describedbelow) can be used to thematically connect two or more characters basedon the knowledge elements. When two or more characters interact witheach other, an interaction group (described below) can be created. Theknowledge base may be generated using NLP core 224 (FIG. 2) inaccordance with the aforementioned rules metadata.

At operation 106, a visual representation of one or more characters'evolution within the script or updated script is generated. That is,various embodiments may present a visualization of the script entered bya user and parsed by frontend analytics engine 206, backend analyticsengine 212, and/or NLP core 224. An example of this timeline isillustrated in FIG. 3E. Compared with the textual representation of ascript in FIG. 3A, FIG. 3E uses a timeline format that reflects theevolution of a character throughout the script.

At operation 108, a storyboard reflecting the script or an updatedstoryboard reflecting the updated script is generated in real-time. Atoperation 110, a 2D/3D preview representative of the script or updatedscript is generated for presentation to the user. In operations 108 and110, some form of abstract or rendered visualization is created torepresent a scene or aspect of the script. That is, the script isconverted from text to visual elements.

A 2D visualization, such as a 2D top-down view of a scene(s), may begenerated using abstract elements representing scene elements, e.g.,characters, props, cameras, etc., although other visualizations infrontend application/UI 202 may be used. A user may “run” a script andvisually observe the resulting scenes as an abstract 2D representation.The visualization can present different aspects of the script inreal-time. For example, as a user is developing or editing a script viaediting panel 204, visualization engine 208 may be receiving, fromfrontend/backend analytics engines 206/212 and/or NLP core 224,information gleaned from the parsed metadata. Visualization engine 208may use this information to generate the character timeline and/or 2Drepresentation of the relevant aspects of the script. It should be notedthat for simplicity's sake, visualization engine 208 may presentabstract representations of the script components, e.g., characters,cameras, props, etc. As the user continues developing and/or editing thescript, the 2D visualization can reflect the current development(s)and/or edits to the script.

In some embodiments, visualization engine 208 may present a 2Dvisualization in non-real time, such as if the user wishes to view thevisualization of the script as developed/edited thus far. Visualizationengine 208 may present a 2D enactment of the script in its current form.It should be noted that script storage 218 may also be used to storedifferent iterations or versions of a script so that visualizationengine 208 may access the different iterations or versions forpresentation to the user. In this way, the user can see visualizationsof the script in different forms or stages during development orediting.

Visualization engine 208 utilizes a visual metaphor language forgenerating the visualizations. Scene metadata (e.g., characters, props,and actions) may be extracted from the script as described above, andused to generate a first visualization. For example, a library of 3Dmodels may be used, where the 3D models are parameterized in terms ofvisual appearance, or other parameters. This parametric representationmay be referred to as a “smart object.” Moreover, one or more catalogsof different kinds of interactions which may be possible between any twopairs of objects, which are referred to as “affordances” may beutilized. Metadata obtained from a script, for example, may be mapped toone or more instances of smart objects and corresponding affordancesthat are being invoked between them. In this way, one or more elementsof a VR environment can be transformed from scene metadata to an actualvisual representation.

Each character in the script (if present in a currently-presented scene)has a position in the visualization. The same holds true for props, andif necessary, equipment such as lights and/or cameras. The position ofthese scene elements can change over time. When users “run” the script,the visualization may dynamically update to show the movement ofelements across, e.g., checkpoints creating an aforementioned 2Dtop-down preview. Checkpoints can refer to a reference time point. Thepreview can be viewed in real-time, slowed down, and/or sped up asdesired. A preliminary timing check can be used. However, the timing ofevents in virtual space can be markedly different from that on 2Dstoryboards because, as described above, different characters, props,interactions, actions, etc., can be present/occurring in the 360 degreespace of a VR environment. Accordingly, one or more of frontend/backendanalytics engine 206/212 and NLP core 224 may adjust any event timing ortiming checks for consistency among different aspects of a scene.

In some embodiments, a “heating map” may be generated. A heating map canrefer to a visual representation of a script element's relationship toanother script element. For example, a heating map can be used torepresent how close a character is to a camera or another character.Information gleaned from such a heating map can provide direction to auser regarding the use of “call to action” hints. A call to action hintcan refer to some cue directed to the audience that draws theirattention to a particular part of the scene, e.g., a subtle increase inlighting that directs the audience's attention to some action orcharacter or other element in that part of the scene. Other calls toaction may include, but are not limited to the use of louder audio thatdirects the audience to the source of that audio. In some instances,heating map information can be used to inform the user or scriptwriterof an aspect of the scene that should be addressed by inserting dialogor having one or more characters engage in some action, etc.

As the user develops or edits the script and/or adjusts one or moreaspects of the visualization, visualization engine 208 reflects theappropriate changes relative to the first visualization. It should benoted that changes can be effectuated by the user interacting with thescript through editing panel 204. Additionally, the user may alter oneor more aspects of a scene using the visualization itself as it ispresented as part of the frontend application/UI 202 which isinteractive. For example, the first visualization may reflect a currentstate of the script as set forth in editing panel 204. The user can thenadjust the placement of objects, props, characters, as well the speed ofthe movement of those objects, props, and/or characters by interactingwith the first visualization. Thus, visualization engine 208 cangenerate a visualization based upon information or input from editingpanel 204, where editing panel 204 may include a section for scriptediting, as well as a section for presenting and/or editing thevisualization.

If the user wishes to add an object that was not identified duringparsing of the script, it can be added manually (again through thevisualization section of editing panel 204), after whichfrontend/backend analytics engines 206/212 and/or NLP core 224 willtrack all references to that object going forward. In some embodiments,frontend/backend analytics engines 206/212 and/or NLP core 224 canupdate the script in editing panel 204 to reflect any objects,characters, etc. added through a visualization and/or not identifiedduring the metadata parsing stage. In some embodiments, a separatevisualization panel or section separate from editing panel 204 may beused for presenting the visualization to the user (not shown).

In some embodiments, persistent data library 220 may contain images ofoutfits associated with one or more characters, sketches (image andvideo), or any other relevant information or data that can be used tocreate and/or edit aspects of the script or visualization.

Generation of the storyboard can be performed by visualization engine208 as well. The process for generating the storyboard is similar togeneration of the 2D visualization described above. However, thecharacters, props, and other elements parsed from the script andassociated with visual elements are reflected in a sequence of stillimages. The storyboard may be presented on editing/visualization panel204. The storyboard may be printed or exported in different formats,e.g., JPEG, PNG, TIFF, PDF, etc.

Again, similar to the 2D visualization and storyboard generation, thescript or updated script may be used to generate a 3D representation ofcameras, locations, objects, etc. present in a scene of a script. Thatis, the 3D preview may be presented using the abstract representationsutilized in the 2D visualization. The 3D preview may be used to assessthe script or story in terms of contextual and/or thematic cohesiveness,consistency, aesthetic appeal, etc. For example, a writer may presentthe 3D preview to a potential producer, or a director may present the 3Dpreview to one or more actors. The director may view the 3D preview inadvance of an actual rehearsal in terms of a live action movie, or priorto animating characters, scene elements, etc. for an animatedproduction. In some embodiments, the 3D preview may be presented to theuser instead of the 2D top-down visualization during scriptdevelopment/editing. In some embodiments, a 3D preview can be output toan augmented reality (AR) or VR-rendering application/device. Forexample, the 3D preview may be presented to a user through previewdevice 222 (FIG. 2). As discussed above, preview device 222 may be anHMD, a see-through display, a video see-through display, a laptopcomputer, a smartphone, a tablet, a mobile device, a projector, amonitor, a TV, and/or other displays.

It should be noted that in some embodiments, the visualization can beused as a basis for creating an actual VR experience based on the scriptor story. That is, a user may associate the abstract representations ofobjects, props, characters, etc. with fully-realized digitalrepresentations, backgrounds can be added, etc. such that thevisualization can be translated into actual media content. Technologiessuch as photogrammetry may be utilized to translate 2D content into 3Dmodels/content for use in the VR experience.

It should be noted that a compiler 214 of backend analytics engine 212can be used to analyze a script for consistency and providerecommendations for addressing any inconsistencies. That is, a textscript, e.g. one entered into or uploaded via frontend application/UI202, can be analyzed for structural and/or contextual validity. That is,a script may be analyzed for any logical inconsistencies. The script maybe analyzed from a “linear” or traditional 2D experience perspective aswell as from a virtual reality (VR) experience perspective.

For example, compiler 214 may receive information regarding the metadataextracted from the script as well as any information or data derivedtherefrom from frontend/backend analytics engine 206/212 and/or NLP core224. Compiler 214 may utilize this information to check a currentiteration or version of the script for structural validity. For example,compiler 214 can determine whether or not any structural errors arepresent. A structural error can refer to errors associated with“physical” representations, e.g., errors in camera placement, propplacement, and the like, that are not physically realizable in a scene.An example of a structural error can be a scenario where a camera “goesthrough” a prop—this is not possible, and should be identified. Compiler214 may also determine whether or not any contextual errors are present.A contextual error can refer to story-based inconsistencies, e.g., theuser has a particular character speaking in a scene, where thatcharacter has not yet made an entrance in that scene, or the characterwas killed off in a previous scene.

Compiler 214 may generate one or more notifications regarding the error.The one or more notifications may simply be notifications that informthe user that an inconsistency exists, as well as where, and/or whataspect(s) or element(s) of the script is involved. In some embodiments,the one or more notifications may be recommendations to the userregarding solutions to rectify the inconsistency. For example, backendanalytics engine 212 may, from the parsed metadata, determine that aninteraction exists between two or more characters in a particularportion of a scene. Compiler 214 may analyze this information anddetermine that a call to action cue should be provided to the audiencebased upon this existing interaction. Accordingly, compiler 214generates a notification suggesting that the user insert a call toaction cue, such as a lighting change or audio cue to direct theaudience to that interaction.

As alluded to above, the disclosed platform provides the ability tocreate and edit scripts, the ability to represent character evolutionusing a visual timeline, the ability to generate storyboards, as well asthe ability to generate pre-visualizations, whether 2D or 3D. Examplesof these functions are described and illustrated in FIGS. 3A-3L.

FIG. 3A illustrates a script layout view with additional featurespresented in the UI to assist in writing a narrative story in naturalEnglish language. An example UI 300, which may be an embodiment offrontend application/UI 202, includes editing panel 302. Editing panel302, which may be an embodiment of editing/visualization panel 204 offrontend application/UI 202, is used to present a textual representationof a script. A user can add, edit, delete, and/or view the script.

A script may include a series of scenes. Each scene can be thought of ascontaining elements referred to herein as “modules.” Modules maydescribe each sentence of the script (1 module=1 sentence, irrelevantwith respect to its written content). A module may be thought of as an“atomic” portion of a story. Examples of modules may be any of thefollowing: location, dialog, transition, or action. A user can describethe module in a sentence, add or drag it into the script or existingmodules. FIG. 3A illustrates representations of these modules. When auser wishes to introduce a module into the script, the user may select adesired module and drag it to a desired point in the script on editingpanel 302.

A location module sets a new environment of the story. A dialog modulesets a spoken conversation of a character. A transition module sets aperiod of changing from one state to the other, normally used at thebeginning and the end of each scene. An action module sets an expressiveaction taken by the characters of the scene. FIG. 3B illustrates anexample of a user hovering on an action module placed into the script.Because the module is an action module, an option to view the action canbe presented to the user vis-à-vis a selectable “film” icon.

Location, dialog and action modules are provided with the option ofchoosing a camera shot. A context menu will open upon selecting a camerashot icon and will give a list of options for each module. For example,FIG. 3B illustrates an example of a user hovering on an action moduleplaced into the script. Because the module is an action module, anoption to view the action can be presented to the user vis-à-vis theselectable “camera shot” icon.

By default, the platform can recommend the most appropriate and commonshot for the given module. Although it is an automated process, usershave the option to manually edit the selected camera shot by choosinganother one from the list. The list may include options for common typeof shots, common angle shots and common camera movements. In the event auser is not satisfied with the given list, an additional option tocreate a new camera shot is provided at the end of the list. This optionallows users to create and assign a name to their own personalizedcamera shot.

Each module, and therefore each camera shot, will create a frame for astoryboard that can be generated. An example of the camera shot list 304is illustrated in FIG. 3C. As illustrated, the camera shot list 304includes a variety of different camera shot images generated for theuser. As noted above, a particular camera shot option in the camera shotlist 304 can be identified (e.g., as being recommended) or is selectedby default), in this case the “long shot.” However, the user may selectanother camera shot from the list, in this case a “medium shot” of thecharacter “Mary.” It should be noted that in some embodiments, thecamera shot may be considered to be a module itself, and a camera shotmodule option may be provided to the user.

In some embodiments, the user can record a soundtrack with their (oranother's) voice over any dialog text as voice clips. Soundtracksassociated with the dialog will be automatically animated in a 3Dvisualization. Characters assigned to the voice clips will be animatedwith lip sync technology so that the character preview (e.g., 3Dpre-visualization) accurately mirrors the speech in the dialog.

FIG. 3D illustrates an example of a record dialog interface 306. Asillustrated, a user may select an option to record spoken dialog of thescript text, e.g., Mary's dialog “What does this mean.” After recording,the user may play back the spoken dialog, e.g., Steve's dialog “Ok, mum.I will can you back . . . .” Time durations associated with the spokendialog can be determined and presented to the user. Visualrepresentations of the spoken dialog, such as a simulated oscilloscope,can also be presented. The user may wish to ascertain volume levelsand/or timing associated with the spoken dialog.

As noted above, the script can be uploaded to the platform. That is, ascript in a platform-specific format or other third-party format can beuploaded. In some embodiments, the platform may perform a conversionfrom a non-native platform format to the native platform format.Additionally, a script can be exported in another formation, e.g., pdf,rtf, txt, etc.

A noted above, a timeline view can be used to illustrate the evolutionof each character in a scene as a graphical 2D representation of thescript. An example of this timeline view is illustrated in FIG. 3E. Thetimeline view can be presented in editing panel 302. That is, a user mayswitch between a textual script representation view and a timeline view.The X-axis represents the different characters and the Y-axis representstime. In order to be fully distinguishable each character may beassigned its own color. The representation of any interactions betweencharacters (interaction view) is created automatically from the parsedscript. The script can be represented in a character-centric perspectivefrom top to bottom. This differs from other timelines in the prior artthat are normally presented horizontally. In this way, a user can switchbetween the textual script view (also vertically presented) and timelineviews and have the position of each module (one after the other) orelements (ordered in time) as a reference, available in both views. Insome embodiments, the timeline view can be presented along with/inconjunction with the textual script view (also vertically presented) sothat both that be analyzed/referenced during script creation or editing.It should be understood that the timeline view can be generatedautomatically. However, the user may still edit the script from a timeperspective.

There are lines and controls for each character, camera, and smartobject in a scrollable container. Additionally, environment sound filescan be placed here as well. By hovering over modules users can getinformation, add new actions, interactions and dialogs, add newcharacters and smart objects, etc. Elements on the lines represent thedialogs and action modules, and can also be dragged into differentpositions and overlapping positions (when various actions or dialogs oran action and a dialog are happening at the same time). Every changemade in the timeline affects the textual script view. For instance, if auser decides to position an element in another position on time (i.e.,positions a dialog after an action), the same change will be reflectedin the script view and the modules will adjust accordingly to the newtime.

FIG. 3E illustrates an example of a timeline view in accordance withvarious embodiments. A timeline may include one or more charactertimelines, an example of which is character timeline 302A. “Bubble” 302Bmay represent dialog to be spoken by a character in the particular scenerepresented in editing panel 302. The user may have entered this dialog,or it may have already been present, and the user is checking to seewhat dialog is associated with a particular character at a particularpoint in time in the scene. Element 302C presents background audio to auser.

It should be understood that the user may manipulate one or more aspectsof UI 300, e.g., the character timelines, by selecting and, e.g.,dragging a character timeline in accordance with the user's desiredeffect. For example, when character timelines are brought together ornear each other, it is indicative of the represented characters engagingeither other through scripted dialog or action(s). Differentrepresentations indicating the presence or exit of a character in orfrom a scene can be presented. For example, a solid timeline canindicate the presence of a character, whereas a dotted timeline canindicate that the character has left the scene.

A timeline controller or pointer 302D may be used by the user to“scroll” through a script/timeline in editing panel 302. The timelinecontroller 302D may be associated with a time depending on where it ispresently positioned.

Transitions are represented in the timeline view as a horizontal block302E, affecting all the elements in the scene during this time. Whenusers don't define a transition, by default, the transition will be acut. If defined a specific transition, the transition will happenbetween modules. The time duration of a transition can also be extendedor reduced.

Situations in which the timelines of all characters come together can bereferred to as macro interactions. All characters/objects have the sameinteraction, sharing either position in space or conversation (dialog).Macro interactions are distinguishable from group interactions, wherethere are characters in different positions or having separateconversations. The timelines can be divided into different identifiablezones to differentiate interactions from each group. An example of agroup interaction is that delineated by zone 302F, which represents someinteraction(s) between the characters Steve and Evan. Another “group”may include the character Margaret.

It should be understood that a character can be in a particular scenefrom its starting point or may enter at any point. Likewise, a charactermay leave a scene at the end of the scene or any time prior to the endof the scene. FIG. 3G illustrates a scene represented in a timelinevisualization in editing panel 302. As illustrated in FIG. 3G, thecharacter, Mary, appears in the scene from the beginning, but leaves atpoint 302G during the scene. On the other hand, character, Evan, whodoes not appear at the beginning of the scene enters at point 302Hpartway through the scene.

An idle mode may be used to describe the state of a character when it isin a scene but not acting or when not engaged in any interactions. Forexample, it may be that a director wants to include a character in ascene, but only in a “passive” capacity. FIG. 3H illustrates an exampleof such an idle mode, where the character, Mary, is in the scene, butnot engaged in any interactions or acting. A modified representation ofMary's timeline, in this case, an indication 302I (at the 8 second mark)of her idle status, and a hash line 302I′ used to represent her timelineduring the idle mode, can be used.

As described above, a storyboard view may be another output generatedbased on the script. FIG. 3I illustrates an example of such a storyboard308. Storyboard 308 may comprise a series of images composed in frames.Each frame can represent a module from the script. Referring back toFIG. 3C, the series of images in storyboard 308 may be made up of imagesselected by the user during camera shot selection (or automaticallyselected/by default for that scene).

FIG. 3I also illustrates a 2D visualization 310. A camera 310A,characters Mary and Steve represented as elements 3108 and 310D, as wellas a representation of a tent 310C is presented in the 2D visualization310. As previously discussed, visualization engine 208 (FIG. 2) may beused to generate visualizations, including 2D visualizations andstoryboards. Visualization engine 208 may select different visual orgraphical representations and/or visual orientations depending on thetype of visualization to be generated. That is, the 2D visualization 310presents abstract elements representative of characters or objects, suchas orbs/circles, while storyboard 308 relies on actual images torepresent characters and objects. It should be understood that the usermay have chosen to associate a particular image with a particularcharacter. The user may access a data store of images that can be used.Options can be provided to hide/show characters (actors) in a currentscene, hide/show the camera(s) in the current scene, and adjust field ofview (FOV) that represents a portion(s) of the scene that a camera canobserve, so that users can plan the positioning of elements.

Also illustrated in FIG. 3I is a UI control panel 312 that can beutilized by the user to switch between controlling different aspects ofthe script or various visualizations, e.g., timeline view, storyboard,2D visualization, and 3D visualization. UI control panel 312 may alsopresent relevant information associated with those different aspects,such as times or frames commensurate with an action or dialog involvinga character. As noted previously, a timeline controller 302D (FIG. 3A)may be used to walk through a script or timeline view. UI control panel312 may reflect relevant aspects of a current script, timeline,storyboard, or 2D/3D visualization, and can update depending on where ina script, timeline, etc., the user chooses to view.

FIG. 3J illustrates an example of the heating map functionalitydescribed above. A heating map 314 can be generated in conjunction withthe 2D visualization 310 (or in a 3D visualization). In this example,color can be used to indicate the closeness of two or more aspects orelements of the scene, however, other methods of indicating closenesscan be utilized. In this example, the heating map indicates arelationship between characters 310B and 310D that should be addressedby the user with, e.g., a call to action. In some embodiments, colors inthe yellow, orange, and red spectrums can suggest levels or grades ofcloseness, whereas colors in the blue or green spectrums suggest a lackof closeness. It should be understood that the heating map can reflectany type of relevant information besides just closeness. For example,action or movement, as well as sound can be reflected in a heating map,where different colors can be used to represent gradations or levels ofreflecting the intensity or amount of action, movement, sound, whether acharacter/object is present, etc. In some embodiments, colors of theheating map can be used to indicate how much information can be consumedby a viewer at a given camera position in the scene.

FIG. 3K illustrates an enhanced view of example 2D visualization 310. Itshould be appreciated that darker portions of each element(character/object) can represent orientation or point of view of theelement. For example, character 3106 is represented as, e.g., “looking”to the right, while character 310D is represented as looking up towardscharacter 3106. Element 310C, which as described above, may be a tentcan be represented as having its opening/door positioned in thedirection where character 3106 is positioned in the scene. This viewprovides an abstract, minimalistic preview of the story a user authorsand hides all detail unnecessary to understanding the spatialrelationships between characters or other elements.

FIG. 3L illustrates an example of a 3D preview or pre-visualization of ascene. In this example, the 3D preview is represented as a birds-eyeview. Characters and objects can be represented by corresponding 3Dmodels/images. Actions and/or interactions are represented usinganimation. Sound/dialog (whether input by the user as noted above orlater-introduced) can also be presented as part of the 3D preview. This3D preview provides a user with an easy and efficient way to graspwhere, how, when, and which characters/objects interact. It should benoted that the camera view may be adjusted in this 3D preview. That is,all perspectives may be considered and generated when the 3D preview iscreated. Camera view may be controlled by a user actuating therepresentation of camera 310A. It should also be understood that the 3Dpreview is representative of the FOV and/or the portion of a sceneactually being captured by camera 310A.

To summarize the user perspective, the disclosed platform providesscript, visualization, and previewing functionality through variouscomponents (FIG. 2) that can operate separately or in conjunction witheach other. In some embodiments, they share defined interfaces, and cansupport additional components/functionality.

In some embodiments, the UI itself is built with web-technology (HTML5,node.js, angular.js and Cascading Style Sheets (CSS)) and thereforemulti-platform compliant. UI-Controls built in WebGL with Unreal Enginecan provide the aforementioned 2D and 3D visualizations or previews.Back-end functionality may be written in C# and contains all logic tohandle user inputs to create a consistent story. External components,like persistent data library 220 and NLP core 224 can support the dataprocessing (NLP servers with their pre- and post-processing of thereceived data, written in Java and Prolog) and can accommodate cloudfunctionality for multiple devices. The data stored in persistent datalibrary 220 may be generated/stored in a project-like structure forconsistency and conformity purposes.

Scene metadata which has been prepared by the back-end logic can bedelivered over JavaScript Object Notation (JSON) interfaces to thecontrols. UI “pages” (i.e., the script and timeline views) and theirassociated controls/other UI controls are represented by their owntypescript classes and predefined layout. Containers inside those viewsare filled dynamically by the data the back-end provides over edge.js tothe pages. The UI Controls initially contain all the models,environments and animations to function; however, the back-end provideslogic to upload models in a predefined 3D format.

Regarding the back-end logic, every input (either by user or by thesystem during reading script file) will be parsed by the back-end logicand it is decided how the data will be processed. Data could be sent toexternal components for consistency check and identify persons, actionsand objects. However, the logic can also decide if this text is onlyrelevant for a dialog/direct speech and only has to be represented witha corresponding dialog module.

All data gathered will be used to generate multiple outputs: a logicalconsistency-checked story for the script- and timeline view, enricheddata for the visual controls to build up the scene preview and alsogenerating requests to external components. Additionally, the back-endwill also feed the interface for the persistent layer.

Consistency checks, identifying characters and objects with theiractions are all based on natural language processing (Stanford NLPCore). As described above, in some embodiments, these functions can behandled by one or more of the frontend analytics engine 204, the backendanalytics engine 212, or the NLP core 224. In some embodiments, thisprocessing is relegated to the NLP core 224 due to intense processingrequirements relying on heavy memory consumption and processing time.This can also reduce complexity on the client-side (back-end andcorresponding frontend UI). Furthermore, the platform (or portions ofthe platform) represented as system 200 (FIG. 2) may comprise otherexternal operations/elements, e.g., metadata processing element 226,enriched with metadata to process even more functions before and afternatural language parsing. For example, metadata processing element(s)226) can be used to identify if a character died during some action, andthe character is not able to continue with a second action.

The platform's persistent layer fulfills multiple needs. Based on a XML-and folder structure, projects are saved and loadable, e.g., at/inpersistent data library 220, they are interchangeable for collaborationand they build the basis for other applications, namely VR and augmentedreality (AR) applications. FIG. 4 illustrates an example platform 400that highlights the platforms external connections andextensions/interactions. FIG. 4 illustrates a representation of thefrontend/backend 402/410 of the platform that comprises theaforementioned UI controls, interfaces and logic (which may berespective embodiments of frontend application/UI 202 and backend server210 of FIG. 2). Also illustrated is an example of an external contentcomponent 424, which may be an embodiment of NLP core 224 (FIG. 2), aswell as a persistent data component 422, which may be an embodiment ofpersistent data library 222 (FIG. 2). Moreover, platform 400 may includean application launcher 428 that can be used to launch othercomponents/applications (e.g., AR/VR components 432/434 below).Additionally still, platform 400 may include a collaboration component430 allowing interactions between platform 400 and othersystems/applications, as well as AR and VR components 432 and 434,respectively. AR component 432 and VR component 434 may by embodimentsof a preview device 222 (FIG. 2) or other system or device to which 3Dpreviews may be transmitted for presentation or foraugmented/virtualized realization.

It should be noted that platform 400 may be configured such that it iscompatible with various operating systems, e.g., OS-X, MICROSOFTWINDOWS, etc., by using an Electron framework. To ensure the ability toallow users and other entities/providers to collaborate, projects arefile-based and interchangeable. This can be expanded to a databaseand/or a cloud share. With robust backend logic, requests can behandled, and encryption of personal data can also be enabled.

The aforementioned use of AR/VR devices or applications can be realizedusing “side-applications” (AR/VR components 432 and 434) that providefull AR/VR experiences using, e.g., head-mounted displays (HMDs). Theycan access the same persistent data and can use the same algorithm forbuilding the project (a consistent story and a timeline). However, theycan provide different ways of interacting with a story so the user is beable to navigate through created stories, to have a series ofinteractions, and to manage the story in general.

In this way, a user can have a very early pre-visualization of a story.Every sentence written (for example describing an action between twoactors) will lead to an automated animation with a correctinterpretation of either predetermined or learned conditions. Thosedetailed visualizations allow a user to focus on future tasks which arefar more user friendly in a 3D preview than a textblock.

The platform 400 manages user input (mouse, keyboard and speech) andsystem input (read existing projects), the logic itself, externalconnections and persistent data. This include but not limited to (asdescribed above): drag and drop standard and custom controls in the UI;just-in-time and on-demand parsing of written text with an externalnatural language component; recording of spoken text to be saved asaudio file or parsed by a speech to text engine; providing libraryfunctions (creating, selecting, storing objects like actors, locations,scenes, objects, cameras) with corresponding suitable actions andmethods such as animations; uploading of user-defined locations asmodels or as 180/360 degree videos; defining camera cuts, movements andtransitions as well as their position & orientation.

In some embodiments, platform 400 is implemented as a Client-Serverarchitecture. This leads to a platform that can run on a user's computerlocally, yet still have the ability to have custom component/modules,data processing and/or even single pages, on a remote server.

The frontend 402 of platform 400 may contain two main views and multiplemanaging pages, two standalone-controls, library-controls and modulecontrols. Additionally, multiple pages and modal forms for manage theproject(s) and creating content. Each of platform 400's views andmanaging pages are represented by a single page and special logic intheir function class. Project-wide logic and controls are available forevery page. The use of node.js, java- and typescript and also HTML5 willprovide the interaction between the user and the views, modal forms andmodular controls.

The UI associated with the script view allows the user to insert modulesand build a script. As previously described, different modules areavailable, e.g., “Dialog”, “Action”, “Transition”, and “Location.” Theuser can drag them onto the script view. The UI determines where themodule will be inserted: above all other, between two existing modulesor at the very end. Embedded controls can include the module list: (asmentioned) with “Dialog”, “Action”, “Transition”, “Location.” Foruploading a script, a control is provided which allows the user toupload a script in a predefined format (text file with labels). The UIwill tell the logic to parse the script and the logic will determineitself the different types (identify dialogs, actions, etc.), what needsto be processed by external components (natural language engine), and soon. For script printing, a control is provided to render a PDF directlyfrom the visible text, either with additional description labels orwithout.

Additional controls which are represented by a software command patternin the logic allow for undo- and redo-actions. A control provides foraudio recording, which as alluded to previously, allows the user toupload audio (to use as environment background sound or directly passedto a speech to text engine) for further processing. Another control canbe implemented to let the user search for characters and words in ascript. Keyboard shortcuts controls can be implemented to allow the userto use different shortcuts to create a new module just below a currentmodule, for example. Filters can also be implemented depending on need,e.g., filtering by character, showing dialog with/without recordedaudio, etc.

The script view provides all the functions of the controls mentionedabove, to send the needed events to the logic with all the neededarguments. Furthermore, it shows different UI styles (labels) to enablefast identification of one or more of the following: where a scenestarts and ends; where a location changes; what camera cuts andtransitions are in the script; and all the modules in different styles(i.e. passive description text in italic, spoken text by an actor withtheir name and color as a label).

The timeline view allows the user to see and modify a story in timewisefashion. Characters, objects and camera(s) are represented by ahorizontal line, while actions, transitions and dialogs of theactors/objects/camera(s) are shown as different expanded lines (boxes)on the corresponding line. These aspects can be fully interacted with bythe user. The user can hover over the boxes which will show additionalinformation. For example, hovering over a dialog box will show thedialog text in the tooltip of the cursor.

Embedded controls can include the following: Add/Edit environment soundwhich allows the user to record/delete environment sound which will berepresented in the UI as a soundwave; Add/Edit actor/object/camera whichallows the user to add already known actors/objects/cameras to the sceneby using this menu. It should be noted that every addedactor/object/camera can be represented by a line but does not have to bemandatory; Add/edit action/interaction wherein if a user hovers over aline, an icon appears when the cursor hits the line (is near) helpingthe user with add/edit actions/interactions directly to thecorresponding user at the given time (determined by the position of theline).

Different modal forms can appear depending on what the user selects(i.e. if the user selects “add action”, the form shows all the availableactions relevant to the add action).

In, this way, the user can adjust or expand a script or even build ascript from scratch. All the mentioned controls above provide theirdesired function (e.g., by sending data to the back-end on aninteraction). Furthermore, certain specialized functions are available.A Drag & Resize boxes functions can be provided with which the user canuse to drag displayed modules (box on line) to a new time, although thebox cannot change the owner (drag from one actor to another).Additionally, the user can resize the box which will lead to a newduration of the module. Custom actions of an actor are alsoavailable—these can be, e.g., “mark as out of scene”, “mark as idle,”“enter scene,” and “mark as non-idle.”

With the aforementioned simple (2D) and complex (3D) visualrepresentation of all the actors, objects and objects inside the scene,very early pre-visualization with different level of details can beprovided to the user. Those levels include show/hide camera field ofview, and show/hide actions such as movement and interactions betweenobjects and or actors. These controls are built with a 3D engine(Unreal) and provided as a webGL object to be synced directly with theback-end as well as the front-end.

Embedded controls can include show/hide heatmap control which makes alayer visible (controlled by the 3D engine) and displays the action(heat) as an overlay over the terrain. The embedded controls can alsoinclude show/hide actors control which shows or hides all actors in thescene, FOV control that shows or hides all the field of views for allthe cameras in the scene. With all these controls, a user can plan nextsteps during scriptwriting.

Moreover, controls are provided that allow the user to zoom with aninput device such as a mouse, and a play/animate control that allows theuser to play a particular scene.

Library controls are provided that allow a user to obtain an overview ofall the actors, objects and locations available in a project. Morefunctions like creating, copying and deleting content (actors, objectsand locations) are also available. Those controls need not be related toa 3D engine or to a webGL build, but may be built with the sametechnology as the main forms (Node.js & HTML5). 3D visualrepresentations may be available as thumbnail images.

Multiple pages (views) can be provided to the user that support a fullcreative workflow, project editing, etc. All pages many have similarfunctions since they allow the user to register objects with all theneeded attributes, validate those attributes, and save them in theback-end logic. All further processing (e.g., storing to persistentdata) can be handled by the back-end.

Back-end logic is written in C# and has defined connectors for thefront-end. Currently, the platform 400 can serve two front ends, e.g.,the main UI, an electron-client over edge.js, and the webGL built overthe same interface. The Back-end functions are divided into services,where each service is represented by an interface (based on a Zenjectdependency injection framework). The application hierarchy starts at acentral class for one instance of the application that handles andmanages project loading, saving and unloading, as well as projectcreation. Both the application and project classes provide events forall occurrences modifying the resources they manage.

The back-end controls following different types of data: projects,scenes, entities (aka objects), looks (all objects have a predefinedlook how the preview should be rendered. This is defined by a primitivewhich every preview engine can determine by itself of how to render,affordance descriptions (discussed below), and (smart) objects (aconstruct for every smart entity with a given set of affordances, adefined name and look). This can also be defined by (smart) actors (anextended smart object but used for actors with a unique color), (smart)cameras (an extended smart object but with a different set of defaultaffordances), and the aforementioned modules that acts as the “buildingblocks” for creating a scene. Each module has a start and end time(which is either be calculated by the logic (through the length of thedialog) or given by the user through the manipulations in the UI).Further still, this can be defined by interaction groups (an objectwhich represents a given group of actors/objects which are joined overtime. The platform gives the ability to have a simplified representationinside a scene even if a lot of is happening.) Furthermore, the back-endprovides access to functions such as: NLP-Client (i.e., sending andreceiving data from NLP core 224 (FIG. 2); a parser (a parsing servicewhich only accepts strings by direct input through the logic itself orthe user and returns an expected result-set (i.e. list of objects,spawned affordances, etc.); and a persistence layer which providesmethods of storing and reading projects and entities.

Regarding affordances, platform 400 uses a built-in tool referred to asan “affordance”. An affordance is different to an action (e.g., pick upa stone). Rather it's a direction that has to be reverted. The owner ofthe affordance (stone) provides a set of affordances over which theowner has control. The user (actor) initiates the action (pick up), butthe owner (stone) controls and executes it. This structure givesplatform 400 the ability to control actions through an easy approach.

As noted above, platform 400 may use external resources, such as NLPcore 224 to process data and user input. To reduce the complexity andthe requirements on the client application itself (memory consumingdatabases), aspects like the natural language parsing and therefore thepre- and postprocessing of those inputs are performed on an externalserver. Accordingly, platform 400 may send requests to the externalresources. A JSON structure uses descriptions such as those illustratedin FIG. 5A to parsing a request message from platform 400 to an externalcomponent, such as a NLP core 224. A parse result message may usedescriptions such as those illustrated in FIG. 5B. Similarly,descriptions used in interaction group request messages are illustratedin FIG. 6A, and the descriptions used in interaction group resultmessages are illustrated in FIG. 6B.

The interfaces of platform 400 may be used to connect frontend andbackend logic. The connection can be made with edge.js, which arewrapping functions that allow logic dynamic link libraries to be calledover the defined edge.js interface. The interfaces can handle differentinputs, e.g., drag and drop inputs from mouse or input device-events,keyboard shortcuts, etc. The interface also contains a container tostore complex objects on both sides (C# and node.js). Therefore platform400 is able to attach frontends which can read C# and all others (with adefined JSON structure). FIG. 7 illustrates an example interface.

It should be noted that platform 400 can manage user inputs and devicehow they should be processed. Written inputs (e.g., when a user manuallyinputs a script) can be sent directly to, e.g., NLP core 224.Furthermore, events from the back-end can also interface with externalcomponents, e.g., when a script is loaded, and process it directly.Moreover, an external application, e.g., AR/VR applications 432/434, canbe launched using application launcher 428. These executables orcomponents may be fully independent, yet access the same persistentdata. To exchange data, platform 400 is able to read project folders andfiles. Different users with different instances of platform 400 (orfrontend/UIs) can collaborate, e.g., via collaboration component 430.

The use of natural language processing has been discussed above and inthe context of the present disclosure can be used to parse a script,infer knowledge/information, etc. Although various forms of NLP areknown, natural language capabilities pose a challenge in inferencinginformation from complex stories. That is, natural languageunderstanding has a long way to go before being able to match the levelof inferencing the human brain can make from reading a story.

Another challenge lies in the ability to analyze, compare and sortinformation extracted from a screenplay or a script in order to providethe functionality described above that allows the user to create/editscripts, generate previews, etc. all while remaining contextuallyconsistent. The field of computational narratives has advanced greatlyto represent narratives with the help of story graphs, but these currentdata structures cannot be formed directly from a text-based story. Otherdifficulties arise when representing relationships between characters toa user in way that the user can easily see which characters arecurrently interacting with each other and appear in the same place inthe story.

In accordance with various embodiments, the use of basic/atomicknowledge elements (also referred to as knowledge bytes) iscontemplated. Knowledge bytes represent the information encapsulated inthe script. Additionally, the implications of comparing knowledge byteswith each other to form pairwise relations among knowledge bytes allowsfor recognition of similar and contradictory knowledge bytes.

Additionally, various embodiments use character systems that allow forknowledge bytes to be associated with one or more characters based onthe characters in a narrative that experience these knowledge bytes inthe form of beliefs and desires. This creates a coherent representationof the possession of knowledge in the story. Moreover, each charactersystem can use logical inferencing on the information that it possesses.With the help of the relationships between knowledge bytes, it ispossible to recognize belief/desire contradictions or similarities amongdifferent characters in the narrative, enabling future work towardsdefining relationships between characters as a combination of intentionsand beliefs.

Moreover, various embodiments enable script creators to add reasoningrules and errors vis-à-vis rule sheets. Rule sheets are capable ofparsing structured natural language text to generate certain categoriesof rules and errors, which can then be used by the character systems,enabling them to make logical inferences on the information theypossess.

Further still, the idea of interaction groups is introduced, wherecharacters interacting with each other in a certain time during thestory are considered to be group. Interaction groups are automaticallycreated when characters perform actions together or a character ismoving from one place to another where other characters are alreadypresent (FIG. 3F).

FIG. 8 illustrates an example architecture/workflow for natural languageprocessing as implemented in accordance with various embodiments. Itshows how the creative content originating from the user is processedthrough natural language understanding, and further broken down to astream of knowledge bytes and fed into various character systems and thescript system. Moreover, the reasoning and inferencing done by thescript system and character system results in feedback in the form oferrors and belief-desire relationships, which is relayed back to theuser. The rule sheets and character sheets act as alternate forms ofretrieval of logic and character background respectively.

Inputs to the user interface are presented in three various forms. Theraw script itself is used to build the knowledge base for the system. Araw script can refer to a representation of the main story plot, e.g.,actions of single characters and interactions between characters, aswell as character dialog. The raw script can be stored in a storyknowledge base.

The metadata such as rule sheets allows the system to understand andmake inferences, and character sheets are another source for informationretrieval. That is, metadata allow rules and/or error definitions to bespecified in the form of a story world knowledge base. Moreover,metadata can be used to add information about characters, e.g.,background information, that are stored in a character knowledge base.

Queries from the user are also a form of input, where the output comesfrom the script system and reasoning system in the form of queries andbelief-desire relationships. Queries allow reasoning regarding theknowledge stored in a story knowledge, story world knowledge, andcharacter knowledge bases to be obtained/ascertained by using aknowledge reasoning system (described below). Input can be analyzedusing NLP core 224, which can also create the knowledge byes, being abase of information flow.

All the inputs provided by users are then analyzed using, e.g., aStanford CoreNLP framework, with extensive usage of Enhanced++Dependency graphs. Actions and dialogs are processed in order to createand save knowledge bytes containing new information. In the case of userquestions, inferences about the stored information are made, and outputbased on the inferences are presented to the user. If there are anyproblems during text analysis, appropriate errors are returned to theuser interface too.

As previously noted, knowledge bytes are basic elements or “atoms” ofinformation being passed on to the system, e.g., platform 400 (FIG. 4).A parser can be used to translate all information into a stream ofknowledge bytes which can then be passed to respective character adscript systems.

The script system stores all the knowledge bytes that are present in thescript, and also in the character systems. It can access eachcharacter's knowledge base and check for errors and relationships basedon similar or contradictory beliefs and desires.

The character systems contain multiple knowledge bases and logicalreasoning systems, e.g., one for each character in a story or narrative.They are responsible for storing snapshots of the story as arepresentation of what each character perceives about the world aroundthem, along with their “brains” in the form of the reasoning systems.These systems allow for character-centric representation and reasoningfor the story as well.

Rule sheets are used to define rules and error definitions for stories.Rules can be provided by users in natural language, after they areautomatically translated into a form used by a logical reasoning system.As soon as there is a query from a user, the logical reasoning systemcombines information stored in the script and character systems withinformation provided in rule sheets to answer questions, show possibleerrors in a story or relationships between the beliefs of differentcharacters, etc. FIG. 9 presents rules created for example sentences,recognizing beliefs and desires of characters and containing informationabout time and location of interactions.

In some embodiments, one character sheet corresponds to one character inthe story and is used to define its background—a history that happenedbefore the start of the actual script. All information contained thereinis added to the corresponding character system and used while askingquestions about the character and its knowledge.

FIG. 10 presents a tool that allows users to create story scripts andspecify rules and error definitions, which may be presented via frontendapplication/UI 202 (FIG. 2). Its layout is similar to that of editingpanel 204. FIG. 10 illustrates an example query in a form of question“Who owns the crown?” that generates a response after analyzing thewhole script provided by the user. In this way, a user can easilyaddress or pre-emptively avoid any thematic or contextualinconsistencies without having to, e.g., manually re-read a script.

FIG. 11 illustrates another example implementation of interaction groups(also described above and illustrated in FIG. 3F). FIG. 11 illustrates afirst interaction group 1100 comprising the characters Mary, Margaret,Evan, and Lucy. A second interaction group 1102 includes charactersSteve and Ronald. It should be understood that interaction group 1100may be formed upon the character Margaret joining the characters Mary,Evan, and Lucy.

The particular details regarding knowledge bytes will now be discussed.As noted above, a knowledge byte can be defined as the smallest unit ofinformation about a narrative. It can represent any kind of informationpresent in a script, such as action, dialog, or questions. Moreover, theknowledge byte also has support to store the location, time point, andthe coordinates where it takes place. This is done in order to processspatial knowledge which can be used for spatial reasoning, which couldbe used to support newer forms of storytelling.

The structure of the knowledge byte is as follows:

-   -   Unique ID—every knowledge byte extracted from the script has a        unique identifier;    -   Time point—every knowledge byte has a time point to keep track        of the point in time in the narrative at which this knowledge        byte was first produced;    -   Coordinates—each knowledge byte is associated with        three-dimensional coordinates to allow for spatial reasoning and        representation of knowledge bytes;    -   Locations—considering that the medium of text is a script,        knowledge bytes can also store the location where they are        created, to allow for location-based representation and        reasoning, as well as location-based questions;    -   Subject—the subject of the knowledge byte; Relation—the        relationship that is being stored;    -   Object—the object to the relation;    -   Relational Modifier—the knowledge byte uses the relational        modifier to store additional information about the relation        (commonly used for prepositions);    -   Relational Object—the relational object stores the object that        may accompany the relational modifier;    -   Relational Question—the relational question stores a question        tag if the knowledge byte is a question, and allows the platform        to look at what question the user is asking;    -   Negation Flag—a negation flag allows the knowledge byte to keep        track of negation;    -   Speaker—when using a knowledge byte to represent information        from dialog, the dialog speaker is stored as part of the        knowledge byte (and can be used to compare characters learning        about similar or contrasting information from multiple sources).

The importance of breaking down the information into knowledge bytes isthat these knowledge bytes represent information in the forms of beliefsor desires, and can be stored across multiple characters' knowledgebases. This linking of knowledge bytes across various characters'knowledge bases allows for interesting and illuminating inferences(discussed below).

The introduction of knowledge bytes as a data structure for handlinginformation about stories enables many useful operations. Theseknowledge bytes can be sorted and oriented based on differentparameters. The knowledge bytes can be sorted by time as well as spaceor location, depending on the amount of information the user chooses toprovide to the system through user input. Alternatively, the knowledgebytes can also be aligned, based on the characters' knowledge. Forexample, the distribution of information across various characters canbe observed.

As noted above, different knowledge bases may be created and used inaccordance with various embodiments.

The story knowledge base is responsible for storing all the knowledgebytes that are present in the script, along with references to thevarious characters as well. Every knowledge byte from the script is fedto the knowledge base for the story knowledge, and this allows forreasoning on the information stored in this knowledge base as well.

The story world knowledge base is responsible for storing all theknowledge bytes relevant to a story world. That is, any knowledge bytesthat relate to rules, themes, parameters, etc. associated with aparticular story world or series can be stored in the story worldknowledge base. In this way, any potential inconsistencies, even acrossstories, can be avoided.

Character knowledge bases are used to form a knowledge base for theinformation possessed by each character, and in order to allow thereasoning system to be able to let each character work on their own setof beliefs and desires in a story world. The benefit of having separatecharacter knowledge bases for each character is that it allows thescriptwriters to ask questions to each character and gauge thedifference in their responses based on the information that thecharacter system possess.

Moreover, each character knowledge base stores knowledge bytes with somewrapper information, which describes the relation between the characterspecific knowledge base and the knowledge byte. The relation-specificwrapper contains information about the character's confidence about theknowledge, the time point that the character learns about theinformation, and flags that denote whether the knowledge byte is abelief or a desire. Feedback from the character's reasoning system isalso stored, in order to be sent back to the UI.

The core of the knowledge reasoning system is the logical reasoningenvironment. Moreover, all the character knowledge bases and the storyknowledge base have their own logical inferencing environment. Theprocessed story world knowledge and the knowledge bytes are fed into theinferencing system. The inferencing system is queried for any possibleerrors, and whenever the user directs a question to the particularcharacter.

Logical reasoning can be used by a character system to perform reasoningcapabilities. With the help of rules, it is possible to make deductionsand extract deeper information from the information that is stored inthe knowledge bytes. Another use for the logical system is to answerquestions, e.g., generating Prolog queries, querying them across variousinstances, and reporting the results back in response to user questionsor belief-desire relationships. GNU Prolog for Java can be used toimplement a logical knowledge reasoning system. That is, multiple typesof Prolog facts can be used to store various parts of the knowledgebytes.

Belief facts store or represent the most important information from theknowledge byte and store it in a character system's reasoning system asa belief. The belief facts take two forms:

-   belief(Id, Subject, Relation, Object, NegationFlag).-   belief(Id, Subject, Relation, Object, NegationFlag,    RelationalModifier, RelationalObject, RelationalQuestion).

Desire facts refer to the same or similar information as thatrepresented by belief facts, except that they are stored in thereasoning system as a desire. They have two forms:

-   desire(Id, Subject, Relation, Object, NegationFlag).-   desire(Id, Subject, Relation, Object, NegationFlag,    RelationalModifier, RelationalObject, RelationalQuestion).

Location facts represent locations or scene information, and referencethe knowledge byte ID. Location facts are formatted as follows:

-   location(Id, Location).

Confidence facts store the confidence level as a floating value from 0to 1, and have the following format:

-   confidence(Id, Confidence).

Coordinates facts are similar to confidence facts, but the coordinatesfacts represent information about the coordinates of the knowledge byte.coordinates(Id, X, Y, Z).

Time facts allow reasoning systems to be built with temporal reasoningcapabilities, and have the following format: timeof(Id, Time).

Additionally, the rules and errors in the reasoning system can be used.They can be added by users through a story world knowledge base. Theformatting for rules/errors can vary (an example of which is illustratedand described above regarding FIG. 9).

Reasoning across knowledge bases involves the comparison of knowledgebytes in the various knowledge bases in order to check whether they havea similarity or a contradiction. Looking for a possible connectionbetween knowledge bytes is essential to make inferences across knowledgebases. Using Wordnet synonyms and antonyms for the relations used inknowledge bytes can be extracted. The knowledge bytes can be compared toform relations between them. Various algorithms may be used to comparethe similarity and knowledge bytes.

A relational factor θ is calculated based on the lexical comparison oftwo knowledge bytes, which is a measure of how related they are. θvaries from −1 to +1, with −1 denoting contradictory relation, a +1denoting similarity between the knowledge bytes, and a θ closer to 0representing that the knowledge bytes may be unrelated.

Time points and confidence measures are also taken into account toanalyze relationships between knowledge bytes. By default, in oneembodiment, a 0.8 weight to confidence and 0.2 weight to time have beenassigned in order to make the impact of confidence stronger than time.In some cases, for some pairings of knowledge bytes, there is an abilityto specify custom values for the weights for confidence and time. Thismay be relevant in cases where one may require different level of impactof the difference in time or confidence of the two knowledge bytes onthe possibility that the two knowledge bytes are indeed related.

For any two knowledge bytes, θ denotes the relational score for therelated knowledge bytes. Moreover, the time and confidence values for aknowledge byte can be denoted by T_(x) and C_(x) respectively. It can beassumed that N is the final time point of the script.

The equation for determining the relationship between two knowledgebytes is represented below.

${{Relational}\mspace{14mu}{Factor}} = \frac{{{{C_{2} - C_{1}}}C_{W}} + {\frac{{T_{2} - T_{1}}}{T_{N}}T_{W}}}{\theta}$

In order to form knowledge bytes, necessary information is extracteddirectly from the script written in natural language. The tool forinteraction with the user can be written in C and is running locally.Scriptwriters are able to provide three different types of input:actions, questions and dialogs, each in a specifically designated typeof the input field.

Natural language understanding in some embodiments may be performedindependently on a remote server running a parser written in Java andcooperating with the Stanford CoreNLP framework, i.e., NLP core 224(FIG. 2). The information exchange between a local client and a serveris performed using a universal JSON format. CoreNLP allows a set ofchosen annotators to be used, and specific properties for any annotatorcan be adjusted according to user's needs. These properties can be setindependently for every query.

While parsing a script different kinds of input can be distinguished byassigning a speaker field to each text input—action for a script,question for a question and a name of a currently speaking actor for adialog.

It can be assumed that users input text in paragraphs consisting oflogically connected sentences and every new “thought” should be put in anew paragraph, i.e., a new action, dialog or question input field mustbe created by the user.

A first step of parsing any paragraph involves applying co-referenceresolution to the whole text in the paragraph—it is focused on assigningreal names of actors/objects/places to personal pronouns (“he”, “they”,“it”, etc.) based on previously analyzed sentences. An example input“Adam buys a new phone and he uses it” would be translated into “Adambuys a new phone and Adam uses a new phone.”

Moreover, a coref annotator can be used, and a coref.algorithm propertycan be set to “neural” in order to use a more precise (but slower)neural-network-based resolution. Alternatively, if speed is more of aconcern, a specific value for coref algorithm property need not beassigned in order to rely on a more basic deterministic system.

After applying the co-reference resolution, text is then split intoindividual sentences, which are later tokenized and single words(tokens) are extracted. Each token is usually related to others, whichis resembled in tree-like constituency and dependency graphs. That is,subjects, relations and objects—so-called relation triples, areextracted. As a base for triples generation, the Stanford OpenInformation Extraction was used in accordance with one embodiment,Alternatively, if many possible relations are missed, a custom pipelinecan be created and used to cope with more complicated sentences. Theyinclude simple, continuous, past and present tenses, and active andpassive voice.

A standard way of creating knowledge bytes involves extracting:

-   -   Subject—a noun—forms subj dependency with relation    -   Relation—a verb—usually a root of dependency tree    -   Object—often a noun—forms obj dependency with relation    -   Negation Flag—boolean—True if relation, object or relational        object has any negation (neg) dependency    -   Relational Modifier—usually precedes relational object and forms        case dependency with it (e.g. “go to someone”)    -   Relational Object—usually forms a nominal modifier (nmod)        dependency with relation (e.g. “go to someone”)    -   Relational Question—described in section 7.3    -   There are some exceptions from the standard way of extracting        components of knowledge bytes. The most common case is while        using a verb “to be”, as it has different meanings depending on        the context:    -   an auxiliary verb (aux), usually the case for continuous tenses,        e.g. for sentence “Adam is going to school” the actual relation        would be “go”    -   a passive auxiliary verb (auxpass), used for passive voice, e.g.        for sentence “Adam was chosen” the actual relation is “was” and        the object is “chosen”    -   a copula (cop), used mainly for describing subject, e.g. for        sentence “Adam is honest” the actual relation is “is” and the        object is “honest”.

It should be noted that an object either may not be set (see firstexample in Table 1) or does not have to be a noun as in the case whenthe relation token and the object are connected by open clausalcomplement (xcomp). An example sentence “Adam does not enjoy going toschool” is then resolved as an extended form of knowledge byte as shownin the second example of Table 1, where the object is a verb “going.”For sentences involving more than one subject, relation or objectseveral triples can be created, each consisting of one subject, onerelation and one/no object as shown in the third example of Table 1.Additionally, adverbial modifiers forming advmod dependency with verbscan be extracted, although they need not necessarily form part of aknowledge byte.

Each paragraph and each sentence in the paragraph are indexed and usedto provide a proper timing, saved as a time point in the knowledge byte.All triples generated from one sentence are assigned the same time andresulting actions are set to happen simultaneously. It can be assumedthat questions and dialogs follow similar structure as actions in a waythat subjects, relations, objects, relational modifiers and relationalobjects can also be extracted.

With respect to parsing questions, the biggest difference is thepossibility to extract question words, such as “who”, “what,” etc. Theyare found by looking for either subjects or objects (similar toactions), while making sure that they are relational pronouns startingwith “wh” and stand at the beginning of the sentence.

Moreover, questions can be asked of or directed to a particularcharacter by starting a question with his name. For example, afterasking a question “Adam, who meets Isa?” or “Adam, tell me, who meetsIsa?” a response about knowledge stored in Adam's character systemtelling us of what characters that meet or have met Isa Adam iscurrently aware can be received.

Dialogs differ in a way that the speaker is set to the name of the actortalking. During co-reference resolution we are additionally matching anyusage of personal pronoun with lemma “I” to the name of the actor. Whileanalyzing character's statements, an attempt is made to add gatheredinformation to the character's knowledge system. This is done in orderto be able to perform/obtain reasoning regarding a character's knowledgeand compare it to other characters' knowledge later when askingquestions.

While analyzing actions, how confident an actor is about what he statedin a dialog can be inferred. This is done by checking whether the actoris using one of the words present in a set of confidence words shown inTable 2. Each word is assigned a specific value in a range from 0(impossible action) to 1 (sure that an action happened). For example, asentence “I think that Wojtek owns the crown” stated by Adam wouldresult in creating a knowledge byte containing a belief “Wojtek owns thecrown” with confidence 0.4.

It can be assumed that all sentences provided directly in script have aconfidence equal to 1.

Moreover, a distinction can be made between beliefs (normal statementsmade by an actor in a dialog or in a script, having a certainconfidence) and desires. The latter are recognized by looking for wordssuch as “want”, “wish”, “need” in a provided sentence, either in ascript or a dialog. As a result, sentences “Adam eats chocolate” and“Adam wants to eat chocolate” would create knowledge bytes with similarcomponents, but the former would be resolved as Adam's belief and thelatter as Adam's desire to eat chocolate.

For desires, confidence measurements are not as much of a concern, sothe confidence may always be set equal to 1.

In order to compare knowledge bytes, it can be important to determinewhether verbs in two compared knowledge bytes are either synonyms orantonyms or none of the two. In order to resolve this issue, a WordNetdictionary can be used. While looking for synonyms, a check can beperformed for all other words belonging to the same synsets as the queryword. While looking for antonyms, the built-in WordNet antonym lists canbe used.

Rule sheets allow scriptwriters to create rules that will enableautomatic inferencing of information about the story and characters.Besides this, error definitions can be provided so that consistency ofthe story can be ensured. A scriptwriter is able to check for possibleerrors at any time during the creation of the story.

Rules and errors can be provided in a natural language and they areautomatically translated into a Prolog language structure. In order toparse rule sheets TokensRegex can be used. TokensRegex refers to aframework included in Stanford CoreNLP for creating regular expressionsover (a sequence of) tokens. A check can be performed for propertiesassigned to single tokens extracted by TokenizerAnnotator—lemma, namedentity and a part of speech. Three main patterns for regular expressionscan be used, each of them looking for specific structure of sentencesprovided in the rule sheet input field:

-   -   Type pattern    -   Inference pattern    -   Error pattern

Regular expressions used for different patterns are shown in Table 3 andexample sentences with created rules are shown in Table 4. A name of Asubject and/or object can be typed in all UPPERCASE letters to make ageneral rule for every subject and/or object (see inference and errorpattern examples in Table 4). In other cases, a rule for a specificsubject and/or object (see type pattern example in Table 4) can becreated.

Patterns can be combined. For example, type pattern inside inference anderror patterns can be combined. Co-references and resolve desires andnegations can also be combined. Moreover, while creating rule sheets,information about location can be included (by including a certaincharacter and a time when the action specified was performed). Anexample of such a complex sentence is presented in Table 5.

TABLE 1 Examples of Parsed Knowledge Bytes Table 1: Examples of ParsedKnowledge Bytes Sentence Knowledge Byte Adam cries.belief(0,adam,cry,null,false). Adam does not enjoy goingbelief(0,adam,enjoy,go,true,to,school,null). to school. Adam and Wojtekeat belief(0,adam,eat,chocolate,false). chocolate.

 belief(1,wojtek,eat,chocolate,false).

TABLE 2 Confidence Words Verb Confidence sure 1.0 confident 1.0 realize0.8 know 0.8 state 0.5 say 0.4 think 0.4 feel 0.4 suppose 0.2 believe0.2 assume 0.2 presume 0.2 expect 0.1

TABLE 3 Regular Expressions for Rule Sheet Patterns Pattern RegularExpression Type (?$subject [tag:/NN.*/]+) [lemma:/be/] [tag:/DT.*/]+(?$type [tag:/NN.*|JJ.*/]*) Inference /if/ (?$condition[!lemma:/then|,/]+) /then|,/+ (?$result [ ]+) Error /show/ [tag:/DT.*/]*/error/ (?$error [ ]+) /if/ (?$condition [ ]+)

TABLE 4 Example Sentences for Rule Sheet Patterns Pattern Sentence RuleType Adam is a human. type(adam, human). Inference If SOMEONE buysSOMETHING, belief(Id, SOMEONE, own, SOMETHING, false) :- then he ownsit.

 belief(Id, SOMEONE, buy, SOMETHING, false). Error Show error “Tooyoung” if SOMEONE error(Id1, Id2, ’Too young’) :- is young and he drinksa beer.

 belief(Id1, SOMEONE, be, young, false),

 belief(Id2, SOMEONE, drink, beer, false).

TABLE 5 Example Complex Sentences while Creating Rule Sheets SentenceRule If Adam is tired on Sunday, then he does not belief(Id, adam, go,null, true, to, church, null) :- go to church.

 timeof(Id, sunday),

 belief(Id, adam, be, tire, false). If Adam does not get a candy, thanhe becomes state(Id, adam, angry) :- angry.

 belief(Id, adam, get, candy, true). Show error “Bad habits” if Adamwants to eat error(Id1, Id2, ’Bad habits’) :- chocolate and he is fat.

 desire(Id1, adam, eat, chocolate, false),

 belief(Id2, adam, be, fat, false). Show error “Not at school” if Adamis a student error(Id1, Id2, ’Not at school’) :- and he is at home from8 to 15.

 type(adam, student),

 timeof(Id2, 8),

 timeof(Id2, 15),

 location(Id2, home),

 belief(Id2, adam, be, null, false). Show error “Not wearing helmet” ifAdam error(Id1, Id2, ‘Not wearing helmet’) :- sings in space and he doesnot wear a helmet.

 location(Id1, space),

 belief(Id1, adam, sing, null, false),

 belief(Id2, adam, wear, helmet, true).

Character sheets contain descriptions and stories of single charactersthat have happened before the actual story has started. They are createdwhile making use of parsing scheme used in actions. Character sheetsallow scriptwriters to add more background information about acharacter, which they may not necessarily use in the script itself.Moreover, each action is added to the specified actor's charactersystem, which allows for information contained in the character sheet tobe used in the character's reasoning capabilities as well.

FIG. 12 illustrates an example computing component that may be used toimplement various features of the system and methods disclosed herein,for example, one or more elements of system 200 and/or 400, a userdevice in which frontend application/UI 202 may be implemented, backendserver 210, data library 220, preview device 222, application launcher428, AR/VR components 432/434, NLP core 224, etc.

As used herein, the term component might describe a given unit offunctionality that can be performed in accordance with one or moreembodiments of the present application. As used herein, a componentmight be implemented utilizing any form of hardware, software, or acombination thereof. For example, one or more processors, controllers,ASICs, PLAs, PALs, CPLDs, FPGAs, logical components, software routinesor other mechanisms might be implemented to make up a component. Inimplementation, the various components described herein might beimplemented as discrete components or the functions and featuresdescribed can be shared in part or in total among one or morecomponents. In other words, as would be apparent to one of ordinaryskill in the art after reading this description, the various featuresand functionality described herein may be implemented in any givenapplication and can be implemented in one or more separate or sharedcomponents in various combinations and permutations. Even though variousfeatures or elements of functionality may be individually described orclaimed as separate components, one of ordinary skill in the art willunderstand that these features and functionality can be shared among oneor more common software and hardware elements, and such descriptionshall not require or imply that separate hardware or software componentsare used to implement such features or functionality.

Where components of the application are implemented in whole or in partusing software, in one embodiment, these software elements can beimplemented to operate with a computing or processing component capableof carrying out the functionality described with respect thereto. Onesuch example computing component is shown in FIG. 12. Variousembodiments are described in terms of this example-computing component1200. After reading this description, it will become apparent to aperson skilled in the relevant art how to implement the applicationusing other computing components or architectures.

Referring now to FIG. 12, computing component 1200 may represent, forexample, computing or processing capabilities found within aself-adjusting display, desktop, laptop, notebook, and tablet computers;hand-held computing devices (tablets, PDA's, smart phones, cell phones,palmtops, etc.); workstations or other devices with displays; servers;or any other type of special-purpose or general-purpose computingdevices as may be desirable or appropriate for a given application orenvironment. Computing component 1200 might also represent computingcapabilities embedded within or otherwise available to a given device.For example, a computing component might be found in other electronicdevices such as, for example navigation systems, portable computingdevices, and other electronic devices that might include some form ofprocessing capability.

Computing component 1200 might include, for example, one or moreprocessors, controllers, control components, or other processingdevices, such as a processor 1204. Processor 1204 might be implementedusing a general-purpose or special-purpose processing engine such as,for example, a microprocessor, controller, or other control logic. Inthe illustrated example, processor 1204 is connected to a bus 1202,although any communication medium can be used to facilitate interactionwith other components of computing component 1200 or to communicateexternally.

Computing component 1200 might also include one or more memorycomponents, simply referred to herein as main memory 1208. For example,preferably random access memory (RAM) or other dynamic memory, might beused for storing information and instructions to be executed byprocessor 1204. Main memory 1208 might also be used for storingtemporary variables or other intermediate information during executionof instructions to be executed by processor 1204. Computing component1200 might likewise include a read only memory (“ROM”) or other staticstorage device coupled to bus 1202 for storing static information andinstructions for processor 1204.

The computing component 1200 might also include one or more variousforms of information storage mechanism 1210, which might include, forexample, a media drive 1212 and a storage unit interface 1220. The mediadrive 1212 might include a drive or other mechanism to support fixed orremovable storage media 1214. For example, a hard disk drive, a solidstate drive, a magnetic tape drive, an optical disk drive, a compactdisc (CD) or digital video disc (DVD) drive (R or RW), or otherremovable or fixed media drive might be provided. Accordingly, storagemedia 1214 might include, for example, a hard disk, an integratedcircuit assembly, magnetic tape, cartridge, optical disk, a CD or DVD,or other fixed or removable medium that is read by, written to oraccessed by media drive 1212. As these examples illustrate, the storagemedia 1214 can include a computer usable storage medium having storedtherein computer software or data.

In alternative embodiments, information storage mechanism 1210 mightinclude other similar instrumentalities for allowing computer programsor other instructions or data to be loaded into computing component1200. Such instrumentalities might include, for example, a fixed orremovable storage unit 1222 and an interface 1220. Examples of suchstorage units 1222 and interfaces 1220 can include a program cartridgeand cartridge interface, a removable memory (for example, a flash memoryor other removable memory component) and memory slot, a PCMCIA slot andcard, and other fixed or removable storage units 1222 and interfaces1220 that allow software and data to be transferred from the storageunit 1222 to computing component 1200.

Computing component 1200 might also include a communications interface1224. Communications interface 1224 might be used to allow software anddata to be transferred between computing component 1200 and externaldevices. Examples of communications interface 1224 might include a modemor softmodem, a network interface (such as an Ethernet, networkinterface card, WiMedia, IEEE 802.XX or other interface), acommunications port (such as for example, a USB port, IR port, RS232port Bluetooth® interface, or other port), or other communicationsinterface. Software and data transferred via communications interface1224 might typically be carried on signals, which can be electronic,electromagnetic (which includes optical) or other signals capable ofbeing exchanged by a given communications interface 1224. These signalsmight be provided to communications interface 1224 via a channel 1228.This channel 1228 might carry signals and might be implemented using awired or wireless communication medium. Some examples of a channel mightinclude a phone line, a cellular link, an RF link, an optical link, anetwork interface, a local or wide area network, and other wired orwireless communications channels.

In this document, the terms “computer program medium” and “computerusable medium” are used to generally refer to transitory ornon-transitory media such as, for example, memory 1208, storage unit1220, media 1214, and channel 1228. These and other various forms ofcomputer program media or computer usable media may be involved incarrying one or more sequences of one or more instructions to aprocessing device for execution. Such instructions embodied on themedium, are generally referred to as “computer program code” or a“computer program product” (which may be grouped in the form of computerprograms or other groupings). When executed, such instructions mightenable the computing component 1200 to perform features or functions ofthe present application as discussed herein.

Although described above in terms of various exemplary embodiments andimplementations, it should be understood that the various features,aspects and functionality described in one or more of the individualembodiments are not limited in their applicability to the particularembodiment with which they are described, but instead can be applied,alone or in various combinations, to one or more of the otherembodiments of the application, whether or not such embodiments aredescribed and whether or not such features are presented as being a partof a described embodiment. Thus, the breadth and scope of the presentapplication should not be limited by any of the above-describedexemplary embodiments.

Terms and phrases used in this document, and variations thereof, unlessotherwise expressly stated, should be construed as open ended as opposedto limiting. As examples of the foregoing: the term “including” shouldbe read as meaning “including, without limitation” or the like; the term“example” is used to provide exemplary instances of the item indiscussion, not an exhaustive or limiting list thereof; the terms “a” or“an” should be read as meaning “at least one,” “one or more” or thelike; and adjectives such as “conventional,” “traditional,” “normal,”“standard,” “known” and terms of similar meaning should not be construedas limiting the item described to a given time period or to an itemavailable as of a given time, but instead should be read to encompassconventional, traditional, normal, or standard technologies that may beavailable or known now or at any time in the future. Likewise, wherethis document refers to technologies that would be apparent or known toone of ordinary skill in the art, such technologies encompass thoseapparent or known to the skilled artisan now or at any time in thefuture.

The presence of broadening words and phrases such as “one or more,” “atleast,” “but not limited to” or other like phrases in some instancesshall not be read to mean that the narrower case is intended or requiredin instances where such broadening phrases may be absent. The use of theterm “component” does not imply that the components or functionalitydescribed or claimed as part of the component are all configured in acommon package. Indeed, any or all of the various components of acomponent, whether control logic or other components, can be combined ina single package or separately maintained and can further be distributedin multiple groupings or packages or across multiple locations.

Additionally, the various embodiments set forth herein are described interms of exemplary block diagrams, flow charts and other illustrations.As will become apparent to one of ordinary skill in the art afterreading this document, the illustrated embodiments and their variousalternatives can be implemented without confinement to the illustratedexamples. For example, block diagrams and their accompanying descriptionshould not be construed as mandating a particular architecture orconfiguration.

What is claimed is:
 1. A computer-implemented method, comprising:presenting a first visual representation of a script based on one ormore elements of the script, wherein the first visual representation ofthe script is indicative of one or more words spoken or an actionperformed by a character in the script; receiving updated one or morewords spoken or an updated action performed by the character in thescript associated with an updated script; generating a second visualrepresentation of the updated script, wherein the second visualrepresentation is indicative of the updated one or more words spoken orthe updated action performed, wherein the second visual representationcomprises a timeline view associated with the character, and wherein aline that represents the character spans at least a portion of thetimeline view to indicate a corresponding portion of time in a scene inwhich the character appears; and generating and updating, in real-time,a storyboard, based on the updated script, wherein the storyboardincludes one or more frame objects of a camera shot selection from theupdated script.
 2. The computer-implemented method of claim 1, furthercomprising presenting a user interface configured to receive user inputregarding at least an addition to or a modification of the one or moreelements of the script or one or more elements of the updated script. 3.The computer-implemented method of claim 1, wherein each of the one ormore elements of the script or one or more elements of the updatedscript corresponds to a location, a transition, or a given actionperformed by a given character in the script or in the updated script.4. The computer-implemented method of claim 1, further comprisingrecording, through a user interface, a soundtrack commensurate with theone or more words spoken or the updated one or more words spoken.
 5. Thecomputer-implemented method of claim 1, wherein a portion of the linethat represents the character is brought near another line thatrepresents another character within the timeline view to indicate aninteraction between the character and the another character.
 6. Thecomputer-implemented method of claim 1, wherein a portion of the linethat represents the character within the timeline view comprises anindication of speech spoken by the character during a correspondingportion of the scene.
 7. The computer-implemented method of claim 1,wherein a portion of the line that represents the character within thetimeline view comprises an indication of a lack of interaction or actionassociated with the character during a corresponding portion of thescene.
 8. The computer-implemented method of claim 1, further comprisinggenerating a visualization comprising a first abstract elementrepresenting the character and a second abstract element representing acamera associated with the camera shot selection.
 9. Thecomputer-implemented method of claim 1, further comprising generating aheating map, wherein the heating map represents a relationship betweenthe one or more elements of the script or one or more elements of theupdated script.
 10. The computer-implemented method of claim 1, furthercomprising generating a heating map, wherein the heating map comprisescolors representing relevant information between the one or moreelements of the script or one or more elements of the updated script,and wherein intensity of the colors reflect corresponding relevancy ofthe relevant information between the one or more elements of the scriptor the one or more elements of the updated script.
 11. Thecomputer-implemented method of claim 1, wherein the camera shotselection is a first camera shot selection corresponding with a defaultcamera shot, and the computer-implemented method further comprising:receiving a second camera shot selection; and updating the one or moreframe objects of the camera shot selection from the updated script tocorrespond with the second camera shot selection.
 12. Thecomputer-implemented method of claim 1, further comprising: generating aproject element associated with the storyboard; and exporting theproject element to an augmented reality (AR) application or virtualreality (VR) application for additional visualization of the storyboardat a device configured to display AR or VR content.
 13. Thecomputer-implemented method of claim 1, further comprising: accessing alibrary of 3D models, wherein the library of 3D models comprisespredefined data objects; and generating the storyboard to incorporate asubset of the library of 3D models, wherein the storyboard correspondswith a 3D visualization of the updated script.
 14. A system, comprising:a memory storing one or more instructions; and a processor for executingthe one or more instructions to: present a first visual representationof a script based on metadata extracted from the script, the metadatabeing representative of one or more elements of the script, wherein thefirst visual representation of the script is indicative of one or morewords spoken or an action performed by a character in the script;receive updated one or more words spoken or an updated action performedby the character in the script associated with an updated script;generate a second visual representation of the updated script, whereinthe second visual representation is indicative of the updated one ormore words spoken or the updated action performed, wherein the secondvisual representation comprises a timeline view associated with thecharacter, and wherein a line that represents the character spans atleast a portion of the timeline view to indicate a corresponding portionof time in a scene in which the character appears; and generate andupdate, in real-time, a storyboard, based on the updated script, whereinthe storyboard includes one or more frame objects of a camera shotselection from the updated script.
 15. The system of claim 14, whereinthe processor further executes the one or more instructions to generatea knowledge base comprising information inferred from the metadata andat least one of rules metadata or user queries regarding the script orthe updated script.
 16. The system of claim 14, wherein the processorfurther executes the one or more instructions to receive user inputregarding at least an addition to or a modification of the one or moreelements of the script or one or more elements of the updated script,each of the one or more elements of the script or the one or moreelements of the updated script corresponding to a location, atransition, or a given action performed by a given character in thescript or the updated script.
 17. The system of claim 14, wherein theprocessor further executes the one or more instructions to generate andupdate the storyboard by creating a storyboard frame based on each ofthe one or more elements of the script or one or more elements of theupdated script.
 18. The system of claim 14, wherein a portion of theline that represents the character is brought near another line thatrepresents another character within the timeline view to indicate aninteraction between the character and the another character.
 19. Thesystem of claim 14, wherein a portion of the line that represents thecharacter within the timeline view comprises an indication of speechspoken by the character during a corresponding portion of the scene. 20.A non-transitory computer readable medium storing instructions that,when executed by a processor, cause the processor to perform the stepsof: presenting a first visual representation of a script based on one ormore elements of the script, wherein the first visual representation ofthe script is indicative of one or more words spoken or an actionperformed by a character in the script; receiving updated one or morewords spoken or an updated action performed by the character in thescript associated with an updated script; generating a second visualrepresentation of the updated script, wherein the second visualrepresentation is indicative of the updated one or more words spoken orthe updated action performed, wherein the second visual representationcomprises a timeline view associated with the character, and wherein aline that represents the character spans at least a portion of thetimeline view to indicate a corresponding portion of time in a scene inwhich the character appears; and generating and updating, in real-time,a storyboard, based on the updated script, wherein the storyboardincludes one or more frame objects of a camera shot selection from theupdated script.